Neural networks (NN) are widely used in automatic speech recognition. In particular, neural networks (NN) may be employed in feature extraction or phoneme recognition. When used in phoneme recognition, neural networks are typically trained with set of training data. During a training phase, the parameters of the neural network are estimated. The neural network with the estimated parameters is then used in phoneme recognition as part of a speech recognition system.